Measuring Effectiveness of Telecom Regulation Using Perception Surveys

The Telecom Regulatory Environment (TRE) survey presented in this paper is a tool to evaluate the effectiveness of the Telecom Regulatory and Policy Environment of a country. It asks senior stakeholders to assess the effectiveness (efficacy) of the regulatory and policy environment in the telecom sector of a country along seven dimensions. The dimensions are based on the General Agreement for Trade in Services (GATS) regulatory reference paper on telecommunications, with some additions. Three telecom subsectors of fixed, mobile and broadband are evaluated separately. The evaluation is done on a Likert scale of one to five. The stakeholders are selected to represent in a balanced manner those directly affected by regulation and policy (such as operators and equipment manufacturers), those observing the sector with broader interest and affected by its overall performance (such as lawyers, consultants, investment analysts) and those representing consumer interests (such as civil society organizations, other government institutions). Equal weight is given to each sub-group of stakeholders in order to ensure that a range of views are represented in the final score. The results can be used to diagnose the positives and negatives of a country’s regulatory environment. When multiple countries are surveyed, resulting comparisons can give insight into best or worst practices and act as tool for leaning. For investors who are evaluating investment options in multiple countries, the TRE scores can act as a proxy indicator for regulatory risk. The paper surveys the literature on measuring regulatory performance in infrastructure sectors, describes the TRE methodology and discusses the results from implementing the survey in 2006 and 2008 in a number of South and South East Asian countries. 1.0 Introduction: Measuring regulatory effectiveness Networked infrastructure industries such as electricity, telecommunications and water have historically been subject to regulatory oversight. Certain characteristics of these sectors (such as extensive 1 This research has been funded through a grant from the International Development Research Center (Canada) and the Department for International Development (UK). The authors also acknowledge the contribution of the research team that helped refine the methodology, implement TRE surveys and synthesize results in various countries. This team (over the years) has included Payal Malik (India), Malathy KnightJohn (Sri Lanka), Deunden Nikomborirak (Thailand), Divakar Goswami (Indonesia), Chanuka Wattegama (Indonesia), Joseph Wilson (Pakistan), Erwin Alampay (Philippines), Miraj Khaled (Bangladesh), Lorraine Carlos Salazar (Philippines). Important contributions made by Divakar Goswami, Sriganesh Lokanathan and Dimuthu Ratnadiwakara in refining and critiquing the methodology are also gratefully acknowledged. 2 [email protected], COO and Indicator Specialist, LIRNEasia 3 [email protected], Chairman and CEO, LIRNEasia economies of scale and scope, large sunk costs relative to fixed and variable components, and provision of services deemed essential) have necessitated and justified regulatory action in order to ensure consumer welfare as well as to encourage private-sector participation. These sectors which were government-owned integrated monopolies in many countries have undergone significant changes through institutional reforms that include vertical and horizontal restructuring, privatization and the establishment of effective regulation. Though experience has varied considerably across countries and sectors, as Kessides (2004) summarizes, for the most part reforms have significantly improved infrastructure performance, as measured by increased investment (leading to increased coverage), service quality, productivity, cost effectiveness and prices that are more closely aligned with underlying costs. The actions of the regulator after such reforms have taken place are also important. These factors are true in developed as well as developing countries. For example, Fink et al (2002) study of 86 developing countries across Africa, Asia, Middle East, Latin America and the Caribbean show that competition, independent regulation and privatization (and the correct sequencing of these actions) produced the most positive effects. Given the importance of regulatory and policy action in the performance of networked infrastructure industries, significant attention is paid to measuring the impact of such regulation. The literature that analyzes, quantifies and compares the impacts of regulation on networked infrastructure industries is therefore substantive, with contributions made by the traditional academic disciplines and researchers within multilateral donor agencies (the latter’s presence in the literature is not surprising because these institutions are involved in funding regulatory reform in numerous countries). Later in the paper we will situate the TRE in relation to other methods for evaluating regulatory effectiveness by scanning the current literature. But first, we describe the TRE methodology and describes how it is implemented. 2.0 The TRE methodology The TRE provides measure of stakeholder perception of a country’s regulatory environment. The TRE instrument/survey asks informed stakeholders to rate (on a Likert scale of 1 to 5, 1 being highly unsatisfactory, 5 being highly satisfactory) the Telecom Regulatory and Policy Environment in a country along 7 dimensions. Seven Dimensions: The seven dimensions are the regulation of the following: market entry, allocation of scarce resources, anti competitive behavior, interconnection, tariff regulation, universal service obligations and quality of service. Five of the seven dimensions are based on the GATS (General Agreement for the Trade in Services) reference paper on telecommunications, the widest consensus globally on what constitutes “good regulation” and adopted by over 80 countries. Three telecom sub-sectors: The three sub-sectors of telecom – mobile, fixed and broadband – are evaluated separately. In other words, the stakeholders are asked to evaluate the effectiveness of the regulatory environment as applicable to the fixed telephony sector along 7 dimensions, the mobile sector along 7 dimensions, and the broadband sector along seven dimensions. So a total of 21 responses are requested from each respondent Three categories of survey respondents: The respondents to the survey fall into 3 different categories:  Category 1: those directly involved in the sector such as operators, equipment vendors.  Category 2: those indirectly impacted by the sector or those studying and those observing the sector with broader interest such as consultants and lawyers.  Category 3: those who represent the broader public interest such as media personnel, other government officials, former regulators and staff, and civil society organizations. Current telecom regulators and telecom policy makers are not surveyed because the goal is not to obtain a self-evaluation, but to evaluate how those impacted by the regulators and policy makers feel. Number of respondents: The larger the number of respondents, the less biased the survey. However, the goal of the TRE is to measure perception among informed stakeholders, those who have expert, indepth knowledge about (or first-hand experience in dealing with and navigating) the various aspects of the regulatory and policy environment in a given country. Therefore the pool of potential respondents is limited to the senior level decision makers in various organizations. For example for Category 1, questionnaires are only sent to (and responses only accepted from) CxO level employees (e.g. Chief Executive Officers, Chief Regulatory Officers, Chief Marketing Officers, etc) at telecom operators or equipment manufacturers. If a team of consultants were hired by the regulator, the team leader is the ideal potential respondent in Category 2. While such respondents do provide more knowledgeable set of responses, they are few in number. But in order to minimize bias within a Category and across the whole, a minimum number of responses have been specified, and without meeting this minimum the TRE survey is considered incomplete. The minimum number of responses per category is 15, resulting in a minimum of 45 responses being required per country. Our experience from implementing TRE surveys in 2006 and 2008 (in 6 and 9 Asian countries, respectively) shows that this is a reasonable target, achievable by a researcher or a research organization with a sufficiently prominent profile within the country. For micro-states (those with less than 1 million population), the above target is impossible. Therefore the minimum acceptable per category is 5 responses and minimum per country is 15. Experience in surveying the Maldives (population of 300,000, duopoly in each of the 3 sectors fixed, mobile and broadband) in 2008 showed it was possible though extremely difficult (specially if the rule of only surveying CEOs of the operators is followed). Equal weight to each respondent category: The stakeholders have different incentive structures, and will therefore have differing opinions of whether specific regulatory actions are “good”, since what is suitable to one party (say, companies) may not be suitable to others (say, consumers). Yet in order keep the result as objective as possible and to avoid over-representing one point of view, each respondent category should contribute equally to the final TRE score. However in these types of surveys it is not possible to control how many completed questionnaires will be returned by respondents in each category. Therefore statistical weighting is used to equalize the contributions made by each category. Administering the survey: The survey (consisting of 21 statements/questions, and space for option comments) is administered through multiple modes: via the internet, through personal meetings, through mail or fax. All responses are confidential and anonymous (only the category of the respondent is recorded, for purposes of calculating above-mentioned weights and keeping track of minimumnumber of response requirements). Attached to each survey is a listing of significant regulatory and policy events that took place in the past year, written in bland language. The purpose of this is to refresh the respondent’s memory. The bland language is used to ensure the respondent is not positively or negatively influenced by the content. Surveys are carried out at the same time in multiple countries in order to ensure that cross-country comparisons can be done with least variations in external factors. A description of each dimension is also provided with each questionnaire. For example, Interconnection is described as and includes “Interconnection with a major operator should be ensured at any technically feasible point in the network. Quality of interconnection comparable to similar services offered by own network. Reasonable rates for interconnection. Unbundling of interconnection. Interconnection offered without delay. Sharing of incoming and outgoing IDD revenue. Payment for cost of interconnection links and switch interface. Payment for cost of technical disruption of interconnection”. The questionnaire, a sample of Significant Regulatory Events from an actual survey, and other documents used in the survey are given in Annex 1. Desk Research and Interviews: In addition to the survey, interviews with experts (stakeholders) and a detailed review of the regulatory landscape (desk research) are done. This enables depth analysis of the TRE scores, in order to identify why scores for certain dimensions are low or high, why scores have changed from previous surveys and so on. Particular attention is paid if the regulatory actions and regulatory framework (as revealed by the desk research) are contradictory to the TRE Scores (e.g. legal rulings on tariff regulation may be reveal them to be “sound” and in keeping with international best practice, but TRE scores for tariff regulation are low). Reasons for such results may be varied – for example, the best-practice regulatory actions may have been taken recently and not yet made an impression on the stakeholders (perception is a measure of cumulative impression over a longer period of time), or the impact of regulatory action may not yet be visible to stakeholders (certain policy and regulatory changes do take time to impact the industry) and so on. The analyzed results are documented in a detailed country-report. Using the TRE results (TRE scores): We believe the best use of TRE results (scores) is as a diagnostic tool, to identify which areas need attention. Country results, when compared historically, can also indicate improvements or declines over time of regulatory effectiveness. As a rule, only aggregate scores per dimension are reported and the scores are not broken down further (e.g., by respondent category). For example, for a given country, the TRE score for Tariff Regulation is reported, even though that score is made up of the scores given by each of the 3 respondent categories. This is to further ensure confidentiality – in certain markets (often in those with a dominant incumbent, few new entrants and a new entrant and a politically charged regulatory environment), revealing Category-wise TRE scores may reveal (or strongly hint at) the identity of respondent or their firms. However, if a researcher is successful in obtaining a sufficiently large number of responses per category, disaggregation may be possible, and may provide for rich analysis. Another use of the TRE scores is to benchmark regulatory regimes in multiple countries. However, different biases may influence respondents in different countries when assigning scores, therefore comparing TRE scores across countries is less defensible, in theory. Yet, cross-country comparisons can be informative and useful as a learning tool. It can also help evaluate regulatory risk, and thereby help the decision making of investors who are examining several countries to invest in. While regulatory risk is not the only concern of investors, it is an important one. Potential investors will often engage in regulatory due-diligence which involves not just scanning the laws and regulations, but also interviews with local experts. The TRE can now quantify the relative regulatory risks in countries ceteris paribus, a country with lower TRE scores in all dimensions poses lower regulatory risk than a country with higher TRE scores. 3.0 Situating the TRE in literature on measuring regulatory effectiveness The literature on measuring regulatory effectiveness is rich and varied. The intent of this section is not to cover every study, but to scan the different approaches taken in various studies, to analyze the differences in their approaches along various dimensions (such as what they evaluate, whether objective or subjective data is used etc) and to identify how and where the TRE is similar or different. We also limit ourselves to the assessment of hard infrastructure industries/sectors. 3.1 What to assess sector outputs or regulatory system? In theory, it is possible to take two completely different approaches to measuring effectiveness of regulatory actions. At one end of the spectrum, is the view that measuring sector performance (along varied dimensions such as increasing coverage, quality and choice and decreasing price) is the best or a sufficient proxy for regulatory performance. After all, if good regulation leads to good sector performance (and bad regulation leads to bad performance), the results of the regulator’s effectiveness is ultimately reflected in a country’s telecom sector’s performance. Yet regulation is just one element that impacts sector performance. As Berg’s (2000) detailed framework shows, industry conditions (including economies of scale and scope), market structure (including vertical integration), historical Experience and many other factors interact together to impact the regulatory system and sector performance. And as Levy and Spiller (1994) concluded, political institutions and economic conditions interact with, and have an impact on, regulatory processes. It may be possible for regulatory actions to be “good” or optimal (i.e. stand up to best practices in regulation when evaluated theoretically), but sector performance be bad, due to negating influence of the larger political or policy climate. The 4 For example in 2006, over 100 responses were received in the Sri Lankan survey. With over 30+ responses per category, disaggregation of the scores of each dimension by respondent category might have been feasible without compromising confidentiality 5 If we were to expand our literature to include studies that evaluate any type of regulatory system (not just limited to infrastructure or hard/networked infrastructure), a good starting point is Berg’s (2009) paper. It analyzes approaches taken by over 8 different studies to evaluate diverse regulatory systems. The TRE assessment is also one of the methodologies covered in his paper. converse (that regulatory actions are sub-par, but the sector itself performs well) is also possible. As is anything in between (for example three of the countries in Spiller and Levy’s study function under “less than ideal” economic and political conditions, but are able to achieve results of good regulation because they at least able to constrain arbitrary administrative actions). Therefore at the other end of the spectrum is the view that regulatory action cannot simply be evaluated by looking at final sector performance, and that it needs to be done through a more insightful or different manner. The TRE methodology is closer to this latter approach, in that it asks respondents to evaluate the overall regulatory environment. Note that depending on the country, some dimensions in the TRE (e.g., universal service obligations) may be under the purview of the policy maker, not the regulator. So in fact the TRE is an assessment of the regulatory and policy environment, not of the regulatory agency. 3.2 Assessment Framework Various criteria or evaluation frameworks have been used by evaluators of regulatory regimes. Brown et al (2006) clearly point out that evaluation of both governance (the formal and informal processes involved in regulation, legal and institutional frameworks) and substance –(the content of actual regulatory decisions) are important. But in most popular studies, the framework focuses on criteria to evaluate regulatory governance. But on the positive side, there appears to be strong agreement among authors on what good regulatory governance entails – most emphasize clarity of assignment of functions, regulatory autonomy, accountability and transparency (Stern & Cubbin, 2005). For example, a NERA (1998) study uses a framework consisting of six elements clarity of roles and objectives, autonomy, participation, accountability, transparency and predictability to assess the impact of governance and regulatory reforms in six Asian countries. One year later, Stern and Holder (1999) used the same data set to further elaborate the evaluation framework (for example, separating the 6 criteria into formal/legal accountability and informal accountability). Noll’s (2000) framework for evaluating regulatory governance includes some of the above components (accountability, transparency), but also include capacity and competency as well. The UK Better Regulation Task Force’s principles of good regulation are a slight variation on the above – the criteria include transparency, accountability, proportionality, consistency and targeting. The TRE Survey asks respondents if the country’s regulatory system is effective along different dimensions, and describes very briefly the elements involved in each. It does not necessarily specify what “effective” means, at least not in any detail. Therefore it is up to the respondent to decide. It is likely that the responses are influenced by the respondents’ experiences with and perceptions of both regulatory governance and substance. For example, consumer groups are unlikely to think a tariff order that increases prices is effective (substance), yet may think that a public consultation process that enabled their views to be heard as effective (process, part of regulatory governance). 6 Performance of the chosen regulatory regime in a country can vary from and A to E or 1 to 5 (E being best possible performance and A being the worst). For example, under the “clarity of roles and objectives”, the sector receives an A if “no specification of any separate regulatory functions or responsibility is apparent, there is no primary law covering regulatory issues, and no effective distinction between policy and regulation” and receives an E if “the regulatory function is well articulated, well enshrined in primary legislation, and separated from the policy and commercial functions in practice”. Above mentioned studies more-or-less converge on a similar framework i.e., close agreement is reached by different researchers on what criteria constitute good regulatory governance. Such convergence is useful because the framework is then made usable to other researchers, who can apply it to different infrastructure sectors globally. In contrast, the European Competitive Telecommunications Association’s (ECTA) Regulatory Score Card for European Countries is designed to specifically evaluate regulatory effectiveness in the telecom sector in one region of the world. The ECTA Score Card assesses countries on adherence to the principals set out in the 2003 EU Communications Framework and associated guidelines and recommendations of the European Commission and European Regulators Group. The assessment is based on three components: 1) the overall institutional environment (scored out of 155 points), 2) the general market access conditions (scored out of 142 points) and 3) effectiveness of regulation and competitiveness in four key access markets and services (scored out of 189 points). It evaluates not just the regulator, but the broader environment – for example the first component (institutional framework) assess not just the regulator but also the legislator, dispute settlement body (if different from NRA) and appeal system. While the ECTA evaluation is comprehensive (the latest assessment contained 118 different aspects of the regulatory environment), its Euro-centric nature is likely to limit its application to a broader set of countries. This is because key evaluation criteria (and questions) such as “timely transposition of the EU regulatory framework” are only applicable to the EU where a common regulatory framework is found – a criteria not applicable in any other region so far. The TRE asks respondents to individually evaluate (by assigning a score between 1 and 5) seven different aspects or seven different dimensions of the regulatory environment are effective. Five of these dimensions (regulatory activity related to market entry, allocation of scarce resources universal service, interconnection, anticompetitive practices) are taken directly from the General Agreement for the Trade in Services (GATS) regulatory Reference Paper on telecommunications. This document largely reflects “best practice” in telecoms regulation. More importantly, it is one that has been painstakingly negotiated and represents the broadest consensus on telecom regulation. At time of writing, 82 WTO member countries had signed up to the regulatory principles spelled out in it. The sixth dimension in the TRE is tariff regulation – added simply because regulating tariffs is one of the primary responsibilities of any telecom regulator, and because regulation of tariffs has a direct impact on sector stakeholders. As such, the 6 (of 7) dimensions of the TRE framework are applicable to most countries. We cannot claim that the seventh dimension (regulation of quality of service) is a global concern yet. It was added to the TRE framework in 2008 because regulators in Asia are finally paying attention to aspects of quality of service. It was a response to requests from regulatory agencies. 7 See http://www.ectaportal.com/en/basic651.html 8 Note that ECTA is the trade body for telecom operators that compete with incumbent former monopolies. As such, the rankings have been disputed by ETNO (European Telecommunications Network Operators association), the trade body representing incumbent operators (https://publicaffairs.linx.net/news/?p=619) . It has also been critiqued by some for being subjective in the manner in which weights are assigned and in the assessment of regulation in the key markets (Weeks and Williamson, 2006; Edwards and Waverman, 2006). Weeks and Williamson (2006) also critique the ECTA scorecard for conflating more regulation with effective regulation without considering the level of competition in the particular market 9 See http://www.wto.int/english/tratop_e/serv_e/telecom_e/tel23_e.htm 3.3 Evaluating Theory (what is written down) vs. Practice (what really happens) Researchers can examine the laws, regulations and orders related to a particular industry and make a conclusion about the effectiveness of the regulatory system. But what actually happens may be quite different to what it prescribed or intended in the rule (or indeed spirit) of the law. So at a minimum, many well-regarded studies will use (local) researchers or experts who have in-depth country knowledge. This is the approach taken by the NERA (1998) study mentioned earlier. The evaluation framework (and associated detailed questionnaire) are applied to each of the 12 sectors across the 6 countries by NERA staff who have previously worked in these countries, or local specialist advisors who worked with NERA staff (page 23, NERA, 1998). The questionnaire primarily focused on questions about the legal framework, but did contain questions that pointed towards practical implementation. Yet Stern and Cubbin (2005, page 15) conclude that because many of regulatory bodies studied were too newly established to have established to have enough of track record, “the results are heavily weighted to aspects of the law and legal obligations relative to actual regulatory practice”. They also point out that “data based on analysis of laws without collection information on practice, are at best, seriously problematic; and, at worst, may be biased and misleading”. A later study by the Prayas group (Prayas, 2003) mitigates such criticisms by first surveying regulators themselves (by sending out a questionnaire to the electricity regulators, covering such issues as regulatory staff and resources, use of consultants, transparency and participation. The information was further supplemented by examination of minutes from regulatory body meetings, scans of websites and so on. The resulting study highlights the difference between theory (well designed regulatory frameworks, established in the past with sufficient time to evolve) and practice (state governments and incumbent electricity companies working collusively to remove the treat of regulation, incumbents who resist providing data or provide wrong data, grossly inadequate staff resources and so on) to be quite significant. The TRE is a survey of stakeholder opinion. But instead of surveying the opinions of regulators, it surveys the opinion of those directly or indirectly impacted by regulatory actions. The respondents fall into three categories that together represent the full spectrum of incentives that are found in the telecom sector. For example, Category 1 (those directly involved in service provision within the sector such as operators, equipment vendors equipment manufacturers) have incentive to maximize profits of individual firms. Within Category 1, the incumbent’s incentives may or may not be aligned with those of new entrants, depending on the situation. Category 3 (those who represent the broader public interest such as media personnel, other government officials, retired regulators, civil society organizations) will often have a very different opinions of what a good regulatory action is. By assigning equal weight to each of the three categories, the TRE attempts to ensure that the final assessment is not biased towards one point of view. By eliminating the level of subjectivity, we argue that the TRE provides a better assessment of a country’s regulatory environment. 4.0 Analysis of TRE scores form recent implementations of the TRE surveys The original TRE instrument was designed to assess regulatory effects on investment (Samarajiva & Dokeniya, 2003). It proposed assessing 2 sectors (fixed and mobile) separately, along 5 dimensions. It was first piloted in 2005 in Sri Lanka (Samarajiva & Dokeniya, 2005). After the pilot study, a sixth dimension was added and the survey was carried out in 6 countries (India, Pakistan, Sri Lanka, Indonesia, Philippines and Thailand) in 2006. The results have been documented in a paper by Samarajiva, et al. (2007). Following detailed discussions with the multi-country research team in 2008, it was decided that a third sector (broadband) should be evaluated separately, and that another dimension (effectiveness of regulation of Quality of Service) should be added. The updated TRE survey was then implemented in 9 countries in the 3 quarter of 2008. Of the original 9 countries identified for the 2008 study, only 8 (India, Pakistan, Sri Lanka, Maldives, Bangladesh, Indonesia, Philippines and Thailand) were completed in time. Annex 1 provides the full set of results (TRE Scores) from 2008 survey. In 2006 and 2008, detailed reports were written for each country surveyed. These reports (Malik, 2008; Wilson, 2008; Wattegama et al, 2008; Khaled, 2008; Knight-John, 2008; Galpaya, 2008; Alampay, 2008; Nikomborirak & Cheevasittiyanon, 2008) analyze the results and give in-depth country-specific information. The purpose of this section is not to repeat the analysis that is already done in the country papers, but to highlight the types of analysis that is possible with the TRE surveys. For example, the TRE studies are an effective method to identify best and worst practices across countries. They are also a good way to track how regulatory action (or inaction) is perceived by stakeholders over time. Several such examples are given below. The graphs shown are ordered by region (South Asian countries on the left, South East Asian on the right). Though the scores of Maldives are presented in each cross-country graph, comparability of scores between Maldives and the other (much larger) markets is questionable. The respondents from the small island nation where most people know each other showed significant concerns during the survey. Even though the market is a duopoly and stakeholders express unhappiness during personal (private) interviews, on paper (in responding to the TRE survey), they rate each dimension of regulation as very effective (i.e. give high scores, close to 4 out of a possible 5, and certainly higher than the midpoint of 3). For these reasons, though TRE scores for the Maldives are presented in the graphs below, they are visually marked with a gray colored box and are not used in the benchmarking or the identification of best and worst practices. 4.1 Best and Worst Practices revealed through the TRE assessments